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TO provide and apply 4 framework for the evalaation of online 
biblioijranhic ret^ ifjval services by determining the relation- 
ships among th^:^ procedures in search request processing * the 
client's gain in knowledcjo, and the degree of client satisfactior 

Type of Study I 

Descriptive 

Sample: , 
148 oearchey requested from the Educational Informatian System 
for Ontario (EISO) conduoted by 1 systems analyst using the 
Educational Hesourees Information Centre (ERIC) database- 
Data collectod May 1976 - April 1077. 

Hethodology: 

A systems framework for evaluation # describing the cycle of 
activities which occur in response to a uearch request , was 
constructed and applied. Data ''on the total sample were 
. collected by a user evaluation questionnaire and a data sheet 
completed by the search analyst* Path analysis was used to 
analyze the relations among process and output variables; 
method cf contact (CONTACT) ; Fystem used (SySTEH) ? turnaround 
times (TURNARND) ; connect time (CNCTIHE) 1 the number ok 
citations printed (CI 7 PRINT); planning, search strategy 
(STRATIME) ; price; a^nd the amount Ifarned by the client 
(AMTLRDTC) * Client satisfaction wai ^measured by 12 items 
.rouped in 6 subseales including: satisfaction with publicity 
materials and directions (Sl)j convenience and helpfulness (S2)i 
timeliness CS3)i quality of technology (54) 1 value of the 
bibliography (SS)j and overall satisfaction (OS). Correlation 
coefficients I means 1 standard deviations* Tablesi figures^ 
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1^ Average delay in running a search was 4.3 days 1 search 
process time was about 1 hour 10 minutesi average 
bibliography length was 104 itomsi approximate cost was 
$28*00; high OS wa s . e xp re s sed . 

2* CNCTIME and STRATIME were positively influenced by perspnal 
contact with the cllenti as occurred in 53% of searches* 

3, AMTLRDTC and OS were not affected by any process variables* 

4* CITPRINT was a significant determinant of price and affected 
satisfaction, 

5. OS was affected by AMTLRDTC and price i si and S2 were 

affected by price* S3 by price, S4 by tURNARND and sySTEM 
and SS by AMTLRDTC, 



Conclusions t 

1, ^ Payment for searches increases the client's expectations 
'Of service I clients' assessment of material pxovidad is 
independent of the amount paid* ' 
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Conclusions. * * 

2, Economiss in services could be made by discouraging 
face to faco incerviews and roatrlcting the length of 
thobiblioyraphy, 

3* Future research is neederl into the determinants of 
knowledge gaii^ experienced by clients* 

SpecialFegturesi 

1 ^ In forma tion Systeni Mo del i fiyure 

2- Path Analysis for Process Model explaining client 
satisfaction j figure 

3, Bibliography 9 iteog 
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ABSTRACT 

Approaches to tho cvaiuation of online bibliographical 
retrieval sorviqes have ta'^en n^rr^^w perspectives and failed to ass^jss 
the interrelationships of various components of tho services. This 
paper presents a compreheneive framewoL^k fur such an evaluation* 
developed using a systems approach. Variables useJ in the framework 
include everts oecurring during a search ryrle, role and status^ 
knowledge and opinion^ and cost* price and demand. Also described is 
the app4 ica t ion of the framework to the evaluation of th,e EducatifjAal 
Inforraation System for Ontario (EIfQ)»s online servica in ord^r to 
determine the relationships between search processing variai^les associated 
with conto and outpuL variaLles measuring the client's gain in knowledge 
and degree of satisfaction. 



/\cconpanying the widesprcc^d intn- :n of on-lijic bibliographic 
retrieval services has, been a rc ^ npiitior the need for their careful 
evaluation. Several approaches i* . been taken to this problem including 
price-dcmaiid analyses, organizational studies ^ data- base ovaluations, and 
end-user evaluations, ^ Hovicv^r, because these stijtlies hav^ t^en relatix'Oly 
narrow perspectives, tliey have failed to assess the inter^ relationships of 
various ccsTqwncnts of the sci^ices in question^ In order to. facilitate this 
typG of analysis, a canprohensiv^ frMCwork for the waluatim of on-line 
bibliographic retrieval serv^ices is needed. The purposes of this paper arc 
to present one such franffiwork, and to provide an exanple of its i^e* 

A systems approadi was used in developing the fr^ework, incorporating 
the traditional categories of input , process , out^t, and feedback* In its 
appHeation, particular emphasis is placed upon assessing the relationships 
between variables of iisterest at different stages, such as input and output, 
Hhile controlling for the effects of inter\'ening variables r It is hoped that 
in achieving a better miderstan ling of si^h relationships, tliose operating 
seareh serv^ices can make more rational decisions' about policies and procedures, 

Systoins Franieu^ork for Ev^aluafion 

In the evaluation fraincH*ork proposed here, a systems a^roacli is used 
to describe the' c>*cle of activities that occur HhenevTer an on-line bibllo^ . 
graphic retrieval service responds to a client's request. In doing so, four 
basic questions mtist be an s^^ red* hhat inputs are required? Khat processes 
tsko placet Uhat outputs are produced? hhat fee^ack occurs at each stage? 
A description of the seqi^nce .of events that occur during a search cycle 
provides ans^vers to tliese questions. 
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Tho search cycle begins witJi the submlssim of a rcqitest by a client 
to the search analyst, a specially trained refci^nco librariaji. They then 

'Negotiate*' the question* imtil it con be restated in a form that the search 

\ 
\ 

analyst can employ to develop a search strategy tMt can be used 'to retrieve 
relev^t citations frOT appropriate data-bases, such as MIC or GNTEKIS,^ 
The aialyst then queries the data-base (s) using a retrieval systcmt such as 
Lockheed Infonnatign System (LIS) 's DIAU)G, System Developront Corporation 
(SDG) 's ORBIT, or United Nations Educational Scientific and Cultural Organisa- 
tion CUNnsCO) 's CDS/ISIS. Uhen conis illations of terms are entcrcds the nimijor 
of relevant citations and a few sample citations are printed. If tJie cita- 
tions are judged to be relevant to tJie eiient's questionp tlie complete biblio- 
graphy is printed off-line and mailed to the requestor. 

The client p on receipt of the bibliography, reviews its contents imd 
makes a |x^rsonal assessment of its relevance and thorouglmess. As a result, 
a request for original docmnents may be made, ishen these are deliwrcd aM 
read, the search cycle, as we defljie It, is complete* 

Fee^ack may occur at almost my stage of tlio searcli c>^cle, Duriiig 
question negotiation the search analyst makes inquiries and suggestions about 
the search topic, and the client TCsponds. Duriiig the retrieval of references, 
a discover)' that an excessive ni^riber of citations have been located j that 
citations are not relevant to the topic, or tliat other tema should be used, 
may cause negotiations to. be reopened and the search strategy altered. Even 

at the end of the entire cvcle, fee^ack occurs uhen an informed and satisfied 

r ■ . ^ 

[or uninfomed and dissatisfied) client returns an evaluation questiortfiaijei ' 

'.his cycle of activities forms tJie system ^wtrayed in Figure 1, At 
each stage, three "actors" are involved: the client, the seardi ar^alyst, and 
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'tiie retrieval system; ^ch of the three is the referent for a ni^er of 
variables \^fiich describe the characteristics of the search senrice's inpjtSs 
the nature of its processes, ^md qualities of its outputs. 

This systems frapiiwork provides a temporal sequence to events occurring 
iirijig a seardi Q^cle, tJiereby allov^^ing the evaluator to make reason^le in* 
fercnces about cause and effect relatljnships. This is importmt, sijice jxslicy 
makers imist have a realistic idea of the ultijnate effects of different decisiOTS 
about inputs Bnd the orgwizatiOT of processing activities. In addition, the 
fr^nework suggests a categorization scheme for variables according to their 
roles as jneasures of inputs, processes or outputs . However, it does not 
provide guidance In the selection of variables to ba measured. For this, we 
turn to the substantive disclplijies of sociology, ps>'chology, and econanlcs, 

Disciplinaty Porspectivos 

The various academic disciplines offer the eraluator a large nimfcer of 
substcuitive theories ^d concepts that might profitably be employed in the / 
selection of x^riables to be .measured and, subsequently, relationships to be 
analyzed, ^ose coiic^ts that we felt wt»re of particular value In evaluating 
a retrieval service were those of role and static from sociolog)*; taiowledge 
fflid opinion frompsychology; and cost, price, and denmd, from economics. The 
first ^*o in the list provide direction m collecting data on a service's 
clients; the secotid tw*o suggest liow one might detennine the effects of a 
sewica on the individual; and the till rd group identifies factors related to 
i servicers fiscal status. Certainly u^er concepts could be selected as 
well, but these se^^en seefned to be adequate since most of tlie variables 
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ropdrted in the literature on search service evaluaticFn could be cOTisidered 
as peasurcs of one of theso concepts. 

The COTicept oi role as it is used in sdciqIo^ is not imlikG the concept 
of role in dr^a*^ The actor, or person, has a part to play and is expected to 
behave in certain way and not others, both by hunself ar^ those aroimd hun, 
i.e., his role set, A person can ox^ct to ha\^ many different roles at 
different tines, but t%TpicalIy plays only one role at a tijiie„3 

By mderstanding the varioi^ roles played by a search ser\'icc's clients , 
one can understand and even predict client behavior md reactions. Since 
requesting a literature search is m act priuarily associated with institu- 
tional as opposed to private life, it is apparent that ^^e individual's pro- 
fessional roles are of prlmaiy interest* 

Sscial static, the second sociological CCTicept used to ^ide the selec- 
tion of variables, refers to a person's stmiding in the CGfroinity and the 
deference the person can expect. It has been Identified as an ij!portant vari- 
abje in the literature on the work of reference librarians since the reference 
inter^'ie^^ of tern re\'erses status positions.^ In a particular instance, a client 
\Aio holds a position of hl^ status may be forced to ainit to being ignorant, 
thereby accepting a tompoTary status inferior to ^at of the reference libra- 
rim. Such a hunbling experience may not be easily accepted by senior profes* 
sors or administraters and their personal reactions might create a barrier to 
the open conimmication of their ^formation needs,*' Ultimately, this could ^ 
affect their lex-el of satisfaction with the service. Hor this reason, toiow- 
ledge of tJie social status of a senrice's clients is of vali^. 

The naturc of toiowledge and t}ie process of learning provide the subject 
matter for learning psychologists v»1io have developed a nLOrfcer of tJieories t^t 



mi^it useful in explsiniJig ktiy an individual rccpests Infoiration md how ^ 
he or she leams frm it. ThQso theories tend to agree m three basic precepts, 
First, a leamer mst t« motivated to loam.. Hiis motivation mv be based in 
the individual's personality (v^iiich opens -up ^othdr realm of psychology) or 
it may be present in the individual's environnipnt, such as the role the person 
plays, . ^ 

^condt a leamer imst -possess sone prior Imwlcdge in order to learn a 
new concept or de\'olop a new understanding, A person alirost fotally laeking in 
Imcwledge about a topic would probably submit im iiyidgquatQ search reqiK^st, 
qijestiOTi Tingotiatlon mi^t prove arduous . and tJio end results mi^t pro\^. less 
than satisfactory. . . " , 

Finally, regardless of whether a single f^t or a complex concept In- 
volving generalizations and inductive leaps is leaTTied, tiie leanier is dimiged 
in the process. , . . ^ 

"ftese three generalisations about the leaming process snp^asize the 
inportance of knowing the extent of a client's motiN^ation, prior kno^^ledgej md 
new toiov*ledge gained as a result of a given search. 

As a search is conducted and the client reviews die results, another 
psychological process ccni^es mto play- .tiie de\iBlopncnt of opinions, (^unions, 
v-liich reveal a prndisposition to act in ^-specific way toward some ■ l (the 
so-called '^attitude object'^, constitute a major set of output variables. Pre- 
sumbly, opinions ^bout tiie quality of a scardi service ^d its products 
detejmine the likelihood of a clieut's returning to submit additional search 
requests. Indeed^ the reciKmiendation tJ'^t client satisfaction be assessed on 
a regular basis is perhaps the most comm recamiCTidation in ^e Vi«rature an 
sea rdi -service evaluation, * 



Tm concepts of costs price, and denmid, 6twm from field of 
economics I also guido the selection of specific vari^kes- Costs are typi- 
cally broken donn into fixed costs (or overhead) uliich represent the dollar 
expense tliat continiies even utien nothing is produced, variable costs, that 
are incurrt^d ^tien production is raised to a given level* For a seardh service 
fixed costs include salaries p equipfrent leases , etc, uiiile triable costs 
are primarily associated ulth the diarges made by coTrq^anies providing the 
retrieval service for connect -timep citations printed, cofiimjiicatlons. 
Costs my be presented as total Ca^rcgate) costs, or in terms of imit costs. 

Setting prices has been a .controversial and difficuit matter for search 
services, first because library services traditionally hav^ been provided 
without diarge and second because of poor data total costs, A related 
concern is the relationship bet^wn price and dcsnmd, WitJiout knffl^ledge of 
tile elasticity of demmd (i'C'P the relative change in demand that occurs in 
response to a change in price) for reaTClies, it is difficult to tell \^1mt if 
mf effect different pricing policies maf'hax^, licnce, measurement of price 
and demand are of critical importance m the evaluation of seardi services , 

A wide array of variables nmist be measured to provide data ne^ssaiy 
for varioia typcfs of econttnic analyses^ Some, such as staff salaries, are 
input variables v^hich TOuld be ijsed to detcnriine fixed costs, O^ers, such 
as the computer connect -time for a g;wn search, arr .process varies les whidi 
i^uld be used to compute variable costs ^id, fx^rhaps, dtftomer prices, Tliese 

i m 

ctota roay^be i^ed in their raw fom for some malyses or aggregated into totals 
for otters. 

In the prtfceedlng description about the way in ^^hich cCTiccpts draw from 
the disci^nes of sociology, psycholo^, and economics can be used to guide 




the selection of variables to be measured in gv^luntijig a search servdcei 
, little was said concarning the lev*ol or t)TJe of aimlyses tlmt could be con- 
ducted. This omnission was pu^oseful, sirice it is anticipated that the 
frainework can be ^plled at different IcvgIs and to explore different rela- 
tionships* In tlie application that follows, data are treated In disaggregate 
form; in contrastp.m dialysis of price-demand would require « aggregate data. 
iUsOy the application is "inter-disciplinary'- in that relationships betk'een 
scVTeral sociological, psychological, and economic variables are investigatedp 
though a discipline-based theory could just have easily been tested. In 
short, tiie level at uliich tiie frMowork is applied and flie particular analyses 
carried out depend upon the specific questions tiie evalmtor or researcher is 
trying to answer t ' 

V^plicatlai 

" The remainder of this paper describes an evaluation of certain aspects 
of a particular on-liiw bibliographic search service, tJia Educational Infer- 
matiOT System for Chtario (EISO) , using the systems franework portrayed in 
Figure 1, The pun^ose of this particular in-^^stigation tos to determine ^e 
relationships among the procedurei follow*ed in processing a search request, 
said tile client's gain in toowledge and degree of satisfaction with the 
results. Because most of the process TOriableSj such as computer connect = 
time, are directly related to costs, the relationships in question have a 
direct bearing on policies and practices involved in operating an efficient 
and eff^tive search ser\'ice^' 



Background . ' 

In 197S the ftitario Ministry of ^Education, faced with a burgCOTiing col- 
lection of odi^atioaial research sti^os ^lich wro effectivnely uriavailable to 
most Ontario educators, coninitted itself to supj^rtijig a vdde rmigijig program 
aimed at the dissendnatlOT of educational rcscardi,j^ As part progrm, 
tw activities involving autonnted services wore funded r The ttitario Edu^^ ; 
tional Rosearch Infonnation System (OTTBRIS) ^hich was to collectp index, 
. and microfiche research re^rts, ^d ^e Educational Informtion System for " 
tOfitario CEIM) nhich ms to provi^ access to existing ^tonmted bibliograj^ic 
^ta bases, Such as ERIC, and the new COTERIS data base. 

The contract for creating and evaluating EISO was awa^fcdjo the Chtario 
Institute for Studies in Educatipn (OISE) , and the service was given space 
witiiin tiie OISE Library, adjacent to the library's jounial collecticn, ERiT; 
mlcroficho 'qol^ctlonp suid normal reference service* EISO was dosipied to be 
a con^lete infonnation service to Cntario's professional educational comniunity, 
supplying access to both bibliographic references and to original documents, 
jiitiaily, only the miC data baserwas searched via SDC*s ORBIT/ At that tljne, 

\ iCTig distance phone-calls to the nearest cffliimmicatiCTi node m ftiffalo. New 
i ■ ' 

York were necossary. Gurrently four search systems are used rei^larly to 

search ^ose data bases most relevant to e^cation, and ^ere is a dioice of 

three different conmnications ne^'orks with l^al nodas* 

■ Paralloling the creation of E ISO itsolf was the development of a program. 

for its regular evaluation orgaJii^ed according to a Viystems ftramework* This 

evaluation has provided regular feedback to the staff involved , and has formed 

the basis for deciding a ni^er of policy issues, such as n^es of advertising 

the serv^ice md meth^s of pricing. 
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Hie major sources of data used in the evaluation are a user evalimtiOTi 
questionnaire and a data^'Shcet con?ileted by the search analyst. In all, 13^ 
different variables are regularly collected for each search that is completed. 
&lecLi'^ of tlicso items mre guided by ^e discipluie based concepts noted 
earlier, i^bst were^ drawn from tiie literature of previous evaltjatioh studies 
^' but only those that were defensible measures of roles, status, toiwledge, - 
©pinion, cost, price, or dcmind ^re included, Uliere gaps appeared, addi- 
tional iteTO were develofwd. - ^ *■ 

Pr^lem * . < 

^0 relationships among the pr^edures involved in processing a request 
- and their iinpact on the client represent a subsystem wi^in the largor evalua- 
~nSr^frameworfe,^imd=.invglye only a fraction of the data collecteU, Key 
process variables ai^ the method of contact CCC^^K^T^the=^ystem^usodJo_^ 
seardi ^e data-bpe (SVSTOOp the length of tijne requiiCd for the search 
interview md to plan tiie search stratogy (STOATBE) , the connect-tijry 
(mCTB^ , tiie nuiAer of citations printed CCITPRIN^ and price (FRIGE) . 
All of these variables except the method of contact relate directly to the 
cost of operating the service or the price charged. The method of uiitial 
ccffi tact, describes tlie actitms of tiie client in playmg the role of requestor , 
If it proves to be an important variable in this mialysis tJien a separate 
investigation using otlier data that have been collected could' bo imdertaken 
to determine why different clients contact the search malyst in different 
ways* Two importmt variables ^ tlie data base being pcarched and the identity 
of the search analyst, have been climiJiated in this list of process vari^les 
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sinCQ they are Leing held constant by limiting tJic aimlysis to Fcarciies of - 
the ERIC data^base conducted by one seardi analyst, . - . ' - 

Tho output or dG^ndent variables at issue are tlie amoioit Iqarncd by 
tiie client about tho topic searched (A^m^RDTC) md his or her satisfaction 
(SATIS), In fact* satisfaction was measured witli twelve items whidi vmro 
grouped to fonn fivQ siAscales (S/\TIS1 to S/\TISS) measuring client satisfaction 
with different aspects of the service; a subset of five of these items were 
added to give an indication of overall satisfaction (S/VnS6) * 

Using tlie temporal sequence of events provided by tJie systems frmnework, 
cm order the,,process and output variables and thoreby refine tho statement 
of the problem into a series of questions about pair-wise relationships* 
Method of contact should clearly be tho first variables folloi^'ed sequentially 
by strategy tijne» tumaroimd timOp selection of a system p eonnoct'-tiTOt cita- 
tions printed^ price, mount learned, mid finallyj satisfaction, Tliis sequence 
does not imp^y that each vari^le causes tJio one that follows; however^ within 
tiie sequences tlie re are a nuriber of pi aO^THle^TOtise -ef f e&t-^palriDgs j _ 

^b^od of contact may have a direct effect on both COTiect-time and 
stratoQ^-tijni since if a person is present in person far more discus sicn ^d 
searchijig is likely to occur than if contact is by telc|^ione or letter* Ccroect^ 
time amd strateg>'"tijnes in turn, could effect the nLorfcer of, citations printod 
smco more time spent on tliese pr^dsly results in a longer bibliography* The 
niinber of citations printed might affect bo^ price, if a variable pricing 
policy is used, and the amomt learned by the. client, sin^ a longer biblio- 
graphy my ccntain more inforaation, .'\nd, botli price and mnoimt learned m^ight 
affect satis factiCTi* 



It is imlikclyJin contrast, that the system used or tumaroimd tim 
will affect any of tlie process variables, thou^ botii might well affect satfs- 
faction:^ tuniaroimd time because long delays mi^t reduce satisfaction, and 
system used bocause one nfight be mom effective tJian tiie other at retrieving 
relevant references* \ l£ tlie latter is the casei tJien the system used may 
also affect the amoisit learned by the . client. Finally, the amoimt a client 
learned niight also be positively iiifluenced by the^tijne spent in negotiating 
aid pluming the search strategy, uidependent of ^e bibliography. 

The problem ^r linvestigation thus becomes testing tiie vAiidity of the ' 
preceding inferonces about cause and effect relationships be^een different 
pairs of vari^les* Tl^e particular method selected for thks purpise is patJi 
rmalysis* 

The paui diagram for thf* relationships described above is displayed iin 
Fi^re 2, Both temporal ^^sequence md cause-effect relationships are indicated 
Dy tile flow of the diagrc^ from left to ri^ht. r^jllwlng tho conventions of 
^th diagrai^, those vari01es ^rith no predecessors are aligned at the left, 
Dje^diagr^^s helpful in showing the flOff of action as a search is executed, 
md in identifying tiie relationships being inVesTigatc^T^The-i^th coefMcleu 
loted on the diagram aro discussed later, but it is worthuhile noting thm 
LJitroduction of h>podietical variables (the e's) which represent extenial 
,^ariables=-w!iose effects are not explaijied by variables in the model, 

\ fbasurcmgnt of Variables v 

■i^ Descriptions of the items used to measuru the vajriables included In the 
>atJi\diagTffln are^listed in Table 1. Variables 1 to 7 aro process variibies; 
he remaijider are output variables. 



TTie first m>'o variables, nicthod of contact (CO^ACT) and system used 
(ffi^roOp are dichotomies and coded as 1 or 0. As a result, care im^t be 
taken in interijreting^ correlation or patii coefficients computed brtween these 
variables and continuous variables. In tJie case of OOP^ACT, a ^sitiTO cor- 
relation bet^i'een it and a cCTitinuous variable would imply that personal 
contact (coded 1) is associated with hi^i values of the oyicr variablCf ajid 
other t>T5es of contact with Iw vali^s, Similarlyi i^sitive correlations 
between SY'STOI and a contintwia vari^le wduld iinply use €f JDC's ORBIT 
(coded 1) is associated wi^ hi^ values of tiie second variable, while use 
of LlS's DLMjOG is associated with ICN values of that variable* ' 

TuniaroiJid time CTVRN/\RVD) is singly the nisiiser of ^ys that lapse 
be^^cen receipt of u soardi request and exectttion of tf;e seWdi. In practice, 
the date of each transaction is recorded, and the difference between the tm 
is calculated at the time data ailg.OTalysed. 

Connect-time in -minutes (Q^CtpE) mi tJio ni^er of citations printed 
CGITORDTD , data listed automatically at ^e end of each seardi, are transcribed 
to the data-sheet mod for EISO's records by the seardi analyst* "Hie analyst ^ 
is also respcnsible foy recording tlie estimated aroiffit of time- spent In ques- 
tion hegotiatiOTi and planning the search strategy (STO/MDE) , Because of the 
cOT^lcx md iterative natuir of tirere^asksr"tlieT-figures recorded for STOATDE 
are probably accurate only witJiin ±15 minjutes* ' * 

The price (TOICE) charged for each search is recorded in dollars. Ishile 
this is straight forh-ardp the pricing policies in effect during the poribd data 
wt*re collected were not* Four different modes of charging uprc i^edi 
1* free seardies all^ated to target groups to encourage trial use of the 
sefeviee, ^ ' 



2, $20 flat rate for clients from a target arca l^atedl in Nor^em Ontario, 
3* $30 flat rate for soarches without m excess ivo nv*er of citations, 
4. $30 plm a surdmrge of $.10 per citation for seardioi mth an exccssiw 
nurtbor of citaticns* ' " 
The ;^licf on charging for an excoisivo nisTter of citations was applied ±n a 
flexible ramer oy tJie'scarch mialystJ T>T)ically, it applied whenever tiie 
combination of connect-tijiio, cojirainicationp md print eDsts exceedsd $45,00,^ 
/ It imist be enq^hasizcd tJiat pricing policy has^a dirbct effect on the types of 
relationship^ ono can esqpect to find. Because most nrsS clionts recelvdd 
seardios for either $30 or nothuig during^ the period imder consideration, any 
effect of PRIffi on o^er variables will ,be InteiT^retablc as the effect of 
charging $30 as opposed to providuig free seivlce. 

The iunount a 'client .learned about his or her topic C^'^^riLRDTC) as a result 
of unde retaking a search wbs tapped by ^e straightforward que it ion listed as 
variable 8. As with o^er one-item variables* it is not possible to coi^te a 
reliability coefficient for this measure. However, ItSuValidity cctifirnied 
by correlating responses on it with questions about the nirter of Driginal 
docments read, and the percentage of mformation that was new to tiie client* 
torrelaticns coefficients were ,36 and .40 reipQCtively, both si^ificant at 
the *0S level, ^ ^ . ^ 

Me^urement of client satisfactim with various aspects of the search 
service were me^ured by five subscales (E\TIS1 to SATISS) consisting of scales 
that included satisfaction with pi^licity nQterials.OTd directions, £^1131) * with 
convenience ^.d helpfulness (SATIS2), with tiroliness of service CSATIS3), with 
'^e quality of tedmolo^ (SATIS4) , ^d with the value of the bibliography 
(SATISS). . ' . 



Individual Items used in constructing subscalos are report©' in AR)cndix 
A, and include thojp used in tho scale used to measuro overall satisfactioh 
(SATIS6). Tho latter seale used five items selected from the subsrAies; not 
all items w^t^ included because many clients had not been .cxj^sed to all , 
asi^cts of tile service and tJierefore had failed to respond to some items. 
Instead, only those items referring to matters to whicJi all:clients had iKieh 
qxposed Kvre includdd in the scale, ^ ^ 

: Reli^llity coefficients for the six scales were ,80 » 32, ^78, ,27, 
.79, and .80 respect i\noly. ^^orall, these coefficients are sufficiently high 
for the scales to be used In, assessing the satisfaction of EISCPs clionts with 
the service , ^ ^ 

^bthodology 7 . , 

\ ■ 

Patli analysis was selected to analyze the relationships arong process 
and output variables because It provides a metliod of testing t}ife validity of 
causal inferences for pairs of varidsles utiile controlling for the rffects of 
otlier variables . ^ In addition, path diagrams provide heuristic portrayals of 
systems relationships whidi are well suited to the systems fraTOHWk used to 
organise the variables in tills study. 

The particular technique used in ^is investigation follws tJie practices 
advocated by Goldberger.^ In particular the path cc^fficlents presented here 
are the stoidardized regression coefficients resulting from rogressicn'bf ^e 
dependimt variola in question on those ^ variables directly affecting it, and 
those alone. In cases uliere there is but one predecessor variable, thtf path 
coefficient is identical to tiie zero-order correlation coefficient; in other 
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casos it is equivalent to the partial°c^v,'^l3tlon coefficient of tte depmident 
variable on the stajidardized prodccossor variable controlluig for other \^ri- 
ahles direct ly affecting the dependent variable, 

Hie ajnoimt of mie.^laijied TOriance m my dependent vari^le_Xi is assigned 
be accounted for by h>pothetical "error'* variables denoted by ej^, ThQ correla- 
tion betu'een md Is equal to 1 » i^"here is ttm multiple= correlation 
betkcen and all predecessor varinbles directly affecting it* 

Not all relotionships. in a path aha lysis are anal vied. First , those 
relationships for hiiich no causal inferences are plausible my tie knitted 
Socondy relationships betVi'eeh exogenoi^ variables (in this case X^ , sX^, mid 
Xj) are typically unanalysed, though tiiey are sometlmDS controlled for if they 
are considered to interact with one motlier. Finally^ correlations among 
the ej^ are assirod to be zero aid renain unanalyied* ' 

Path analysis requires three principal assmiptions vvlii^ may or mav not 
be valid. First, all relationships arc assumai to be linear; second^ rclaticb- 
ships are assLmed to be additive; mid third, it is assumed no interaction 
multiplicative e^fgcts) exists mong variables, As Veils some authors would . 
hold that the variables are assisaed toT>e measured on a ratio scale with sonve 
fixed unit of measure. = 

For the dichotomous variables COCNT/\CT and SYSrap and those measured m 
stmi^rd mits of measure (TUKmND, C^CTDEp Sm\TBIE, CITPRTNT, , and PRIffi) 
nonfe of ^he assuntptions wuld be violated, ^ose measured on Likert scales 
(A5rrLRIfrC and SATIS) may violate assunptions as to linearity and unitary measure- 
ment. However, because mDnotpnlCf if not linear p relatiohs are expected for 
all relationships involving titese vari^le.ip violation of tlio first assunption 
would, at worst, euuse the relationships to be imderestiinated, 

■ . a . J' ^ 

B u 

a 
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The population for this study is the miverseof all 371 searches reques- 
ted from the nducatijnal Informtion SystDm for Qitario between hfay 1, 1976 
and ^ril 30j 1977. Though some data were available on all searches, ^le 
pfepfit^amalysis-was- restri^ been tm on WM£ and for 

lAich the requestor imd returned an evaluation questiormaire* The .total 
nim^er was 148 (40 percent of all searches) * but mitiing resixmses m some 
varif4=£>les result in a smaller ntmijer being used for sane analyses. 

Virtually all searches were placed by clients from the professional edu-. 
cation conrmity in ftitario, which EISO was designed to serve, In^enoralp 40 
percAit of EISO's 'clients mctb from school boards^ 40 percent were from OISE, 
10 percent from other diversities md faculties "of education, and 10 percent 
miscellaneous.. Aj^rnximately 30 percent of, the clients reported ^t they 
were engaged in administration p 17 porcent in teaching, 12 percent m research^ 
18 percCTt in graduate study, 7 percent in librari^ship, and 14 percent ether. 

Findings 

In this, section we shall prwecd fran the general to tfte specific, pre* 
senting first the findings for the path analysis in which over^l satisfactlOTi 
(SATIS6) is "the dependent or output vari^le, and tlien the results for the 
five satisfaction si^scales. > ^ 

Table 2 lists the correlation. coefficients, means, and standard deviations 
for the nine variables that appear in the path diagfan in Figure 2, Several 

of the means need interpretation to be understood. For OQOTACT, X ^ ,S3 

, . . - ... 

implies S3 ix^rcQnt of all clients contacted the search analyst in person; 
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siinilarly» ^ie mean of *19 for ^STB! implies only 19" percent of all searchns 
were done vming Ste*s ORBIT while the remaining 81 perr^nt were carriod out 
using LIS's 

The average df lay in ruming a search was 4,3 days, but the actual search 
^process took about m hour 10 vlnutes, including 46 minutes for *intefvlewing 
aid pi mining the strate^, and 24 minutes for nmnlng the search, Tt^ average . ^ 
bibliography inciuded 104 items, and cost the client $28, though this average 
is of iquesticnable use since most cli^ts either received the search free (9 
percent) 01* paid $30 (82 percent). The average client leaded '*something" 
about the topic searched (since at. 2*2 the mean for A^frMTTC is near tiie 
middle of tlie three-poijit scalfe) and expressed hi^ overall satisfaction (2.6 
on a scale of 3) * 

wilj not malyze tiie correlatiOTS iji detail since that is the purpose 
of the path analysis* HcA^revcr, we would note that initially, only the moxmt 
learned about the topic has a significant zero -order correlation witli overall 
satisfaction* -\ . 

Path coefficients for malyzlng.tho ^usal effect of tte. seven process 
variablos on the ^"o output variables, A^^l^rrC md ^TiS6, are included in 
Fi^re 2, vtfiere each causal path is iridicated by an arrcft'f* The most iJiformative 
paths are those that cm be traced from CONTACT through D,*CTBE md STOATM to 
CITPRDJT, A^f^m^Itp and TOCE, mid finally to SATISa/ ^ ' ; ^ 

Both GslCTM and STOATM are ppsitively mfli^ncod by iJie porsonal con- 
tadt with the client, but the effect of this an the nisriber of citations^rinted 
is. relayed aLiiost solely throu^ ccmect'-time. The ajnoimt leained about tiia 
topic seardicd (A^m^TC) .does not appear to be affected by ^y of the process 
variablos, Tho small negative ecsfficlent between it and SYST^f, uhidi mi^t ; 
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ijTply^that SDC wa^ less effecti\^ thai LIS our search analyst fervently 
believes), is not statistically si^ificmt. Smilarlyp the positive coefficient 
from CITORINT* wlilch wwld ijnply more is leanrted from long bibliographies, is 
not statistically sipiificant, ' 

CITPRINT is a statistically si^ificant determinant of PRIffi as would be 
expected given the pricing policy in which cjiarges for an '^excessive" nL^er 
of citations are made. Other variables uhidi might affect price, sudi m being 
in me of the tar^t groups receiving free searches, are excluded from ^e 
model but are represented by e^, 

Fiimlly, five causal variables ijrqsact upon overall satisfaction (SATISO) 
but only two of these relationships are. statistically si^iflcOTti those for 
/^frLRDTC and TOICE. The first h^^a positive effect, witJi those clients-who ^ 
learned more about their topic having a more positive viewpoint. But^ con^ 
trolling for CITPRINT, A%frLMTC, SYSIBf, and TURMARS^, those vAxo paid a hl^er 
pr^ice were less satisfied overall. That is > all other ^ings being equal, the 
client yAio paid a hl^er rate would re^rt lower o^wall satisfaction tiim one 
who paid a lower rate* Thus, ^ough price did norf^j^ar as an inportant 
variable in the sem-order correlations, it does a^sear as a datennin^t in 
the final analysis* Of the remaining variables whlcli we thought nd^;. affect 
overall satlsfactlai^;iamoly TLTOWID, ffl'ST^f, and CITTREfT, a possible rela- . 
tionship is indicated only for the last. Taking the tiiree brandies from 
CIITRLNT together, it might be suggested thaw it does affect satisfaction, wltj^ 
its positive effects being felt botii directly md indirectly via ^e path 
through amoimt learned, and its negative effects felt via tiie path through 
price* ' . , , 
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Table 3 lists 'tiie path^cocfficients for the -process models m vrfiidi 
not only overall satisfaction Is the dependent vari^le of interest^ but also 

siiscales measure specific characteristics- of the search •service , SATISl 
tiirou^ SAjISS, In t^o JablCp patiis are*najned with the effect vari^lo first 
and the cat^^ variabJo second. The #ijD|cijipts also reflect this ordering^ 

Pm is tJie coefficibnt measuring piQ effect of Xj (OOOTACn oniCj, 
(OICTBE) • An exceptioh to, this wle ^curs when the correlatiOTS remain 
unamlyzed, ai between the pairs of Exogenous variables Xj^to X3* For tiiese 
variables s the zero-order corrolation coefficiehts are presented* 

teans, standard deviationsp ^d saj^le si^es are also given m Tgble 3. 
Sample iiies vary according to tJ^ satisfacticn siAscale being cc^iidered 
since not all clients were exposed to all facets of the service, and hence 
did' not respend to sobe evaluation ^estions. 

Path Coefficients for the first eleven pa^s for tiie five satisfactim 
si±scales are* in fact, estimates of the same path coeffiGionts already con- 
sidored in ^e diagram explaining overall satisfaction* Any variation from 
earlier estijnates is due to variation in tte samples m whidi the estimates 
are based, irf factj the new estimates are quite similar to the earlier OTes, 
except for the sair^le size of 31 on which estimates for path coefficients 
leading to SATIS3 are based. This is to be e^cpected given ^e small sanple 
size, ^d serves as a warning against placini too mudi confidence ih estimtes 
drawn from this siibsan^le. ' - ^ 

The last five coefficients in each coIioti of Table 3 do present new.pa^ 
coefficients whidi measure the effects of ^'STOf, TURM/W©, GITPRDTr, PRICE, 
and j^rTLROTC on the satisfaction subscales, * 
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Startiiig, with SATISl, the degree of satisfaction vfith publicity materials 
afid dirbctions,' we note tliat only PRIffi seems to have an effect , ^d, as beforep 
tJie effect is negative* A siroilar ccsicli^iOT hulsb for S\TIS2, tiie con\^nienca 
of the senrice and helpf^ness of its staff, 

CITPRINT* PRite, AhrrUWC all have si^ificemt effects on tiio satis- 
faction of tJie 31 clients.*^© responded to S\TlS3 uhlch re^ured the timeli- 
ness of the servicep including d^tront delivei^' from wd EISO, fliese 
coefficients are similar to those rwted for overall satisfactioii,' but are = 
someviiiat longer* 

In contra t to the preceding case* only TOR^\Rh©p and SYST&! apfmrently 
affect S\TIS4p which ta^ed client satisfaction witii ^e quality of the 
tedinolpgy^ as measured, by the length of tlie bibliography, readiBility of 
microfiche i and availability of microfidie refers. ' ' 

Satisfaction witii tho ralue of tiie bibliogra^iy ^Ad notorial s located 
CSATIS5) was apparently affected by only ono rafiableg the amount leamed^out' 
the topic. It Is not^^e^^^ati for this specific attitude object the value 
of the bibliography^-OTd nate rials located through it — tlm negative effect 
tJiat price had on otiier types of satisfaction disappears , 



Discussion . 

How successful AS tiie process mo^l.Et explaining client satisfaction? 
We are enfcarr^sed to admt, having put considerable effort into its develop- 
ment md testing, that it is only ^rtly effective at doing tJie jcb for lAich 
it was intended, Fortiffiately» it does provi^ additiwial info™|icn that 
(increases its o\^rall usefulness. 
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As far as satisfaction is concemtd, only mo variable seems to have a = 
GOTsistent effect, mmly price. The negative , effect of price en satisfactien 
was apparent with overall satisfaction, and satisfactiOT wi^ particular com- 
pOTcnts of the servicoi ^jblicit>o materials arid directionSr ^COTVenience and 
helptfulness , and timeliness. It did not affect iatisfaction with the tech* / 
p polo©^ or naterials produced. In intcrprcting this differential offect of 
price, it must be recalled that 91 of all users, received free searches, and 
most ethers paid $30 or more. It would appears itJien, that when individimls 
are payijig for bibliographic seardi, they ft^ct more in the Way of service 
than tiiey would if they were receiving it for free*. Yet, it is iji^rtant to 
note that their expectatioris witJi regard to service do not cairy r^'er to their 
^sessrent of the value of the goods, Khich they apparently assess independently 
of ampurit paid, ^ ^ ' . 

Vh are at a loss to accotmt for the three other sl^ificant relationships 
Sem^n process variables ^d satisfaction scores (SYSTOl imd IUR^l\HNO with 
miality^of 'technology, SATIS4; aid COTRE^T with tiinel mess of service, SATIS3), 
We are inclined to ascribe tiiem to momalies in the'snaller sanples mvolved, 
or to ektranaoi^ variables not in the amlysis* . . ^ 

There is^ of course* another variable, lAidi is related to rarious satii^ 

faction scorw with som&^ consistency , tlie amount learned about the topic T 

. • ^ " *_ • 
searched (X^^riMrrC) • . This variable is an output variable in its own ri^t, 

but intervnenes be,twten satisfaction and all process varl^les except price r 

It is positively delated to overall satisfaction, satisfaction with tiiniluiess 

of service, and most iriportant, satisfaction witii the value of the bibliop'aphy 

and materials. This iast relationship helps to validate tiie satisfaction 

items, since me woiid expect a person v^ho learns more than another frmt a 
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bibliography to be more satisfied with it. But nore ijnportanti it highli^ts 

tho idea that the client is motivated ta learn, and that if this learning 

does not occur, dissatisfactiOT will result. ' = , ^ 

Though the amount learned has proven to be m ijnportmit ijitervenmg 
varii^le explaining client satisfaction, none of the pr^esf^ variables helps 
to account for the extent of learning that t^es place. This failure to 
accomt for variation in learning is an in^rtant finding in and of itself , 
sijice tile following possible causes are ruled out* the system used, nurfcer of 
citatipns printed, md intGinriew/strate©^ tine. It would ap^ar that effllana- 
ticns niist be sought hi tiie personality of knowledge of tiie client, the inter- 
action between the client and searcJi analyst, tiie quality of the search 
strate^', or the topic beijig searched, 

RelatiOTships for the reinainder of the process model have been cOTfirined* 
Price is affected by the nimiber of Citations printed; the niflifcer of citations 
printed is affected by conrrrct-tijTie but not intewiew mid strate©^ tline; 
and both ^^^lect-tljne and interview/strategy tiro are affected by tlie method 
of cOTitact, If tiiere is a^surprise here, it is^that while the personal con- 
tact is responsible for taking a- considerable Momit of tiia search analyst's 
time, the production of Iwiger bibliographies ^d hi^er prices, it seems to 
bear no relationship to either th& airount learned or the level of sat is faction* 

V Inplicationi_ 

'Hiere are two findings in tJiis study uhidi seem of imiiediate ij^rt to . 
the f iardi service mana^r, and me with ijTplications for fuxxm jresear^. 

First, it is apparent that any search seivice Hhidi charges for its 
product nust enphaslze the ser%dco as well as tJie search* It is not enough 

22 ' 



to provide customers with a bibliography and matGrials, but it mmt be done 
in a way that nukes the client realize he is paying as nuch for tiie setvic- 
as for product, and tiiat ho is getting his money's worth, rndeed it can 
be argued that clients mly pay for the service, since ^^iiat is paid is related 
not to tlie ultimte wortJi of tiie material, but to tJie cost of its retrieval. 

Second the finding that process variables ha\^ no clear relationship to 
client sat is.f action or the amount a client leamSs suggests tiiat, where neces- 
saryi economies ray be possible by discouraging face-to^face mterviewi 
(yiicli take more tiro) ^d by restrictinj the lengtJi of bibliographies as 
much as possible. Theso recomendaticmSj if ijnposed m a staff, would pr^ably 
fail (we sper.k from e3q)orience) . But for, search analysts who are responsible 
for their ohh budgets and hfio must defend their existent at least in part on 
the basis of Uie nwriber of clients SQTvmdf it/jmy be encouraging to note that 
it is pn^ably possible to cut a few come re without harming ^e quality of 
the service. 

Finally^ the need for future ^seardi into the ^teminants of the gain 
in taiowledge experlencpd by clients as a result of blbliogra^lc searches Is 
app^arcnt, given its strong relationship with client satisfaction, hhile 
libraries and infornutiOTNs<^tres^havo truditloially defuied their roles as 
providers of informatim, with no guaranteb that, their clients benefit from 
this infonnation, it appears that for isearch services to succeed in satisfying 
their clienty noeds, a better imycrstandlng of the persoijal tmJ psychological 
factors Ijivolved i^^i^dcd* In particular * if it is ^ssibleHto est^lish in 
ad\iince the likelAood of a seardi's ruccess in filling a client's needs, then 
search analysts may be able to better distinguish be^^een those clients ^^'ho 
are likely to profit and those v,1io are nut. 



Limitations 



' The suggestions above wore presented witiiout qualification, but it should 
be oi^hasized that they are made within tJie context of a single search sen^ice, 
ETSO, sendng a special population, the professional educational coitimmity m 
Qitario, Quiada. Gi\^n the nature of its client grQi^; EIM*s staff primarily 
cmdsjcts in-dapfJi, retrosj^cti^'e searches requested as a basis for researdi 
reports or back^oisid studies. The length of search interviews, connect -tinfc, 
and tJl^ number of citations printed tend to exceed norms reirortcd in the 
literature, but _^are not, u*© feel, out of line witii the task.® Never^eless, 
.given niSO's notms, it may well be tha^ it cm cut cdrners witAout damaging f 
the quality of service, whereas aiother seivice could not* Therefora, tJie 
conclusions and recomnendat ions made here should not be considered as conclu- 
siCTis of general applic^ility. 

Ihe systems frarowork for evaluating an on-line bibliographic search 
service described in this paper was successfully applied to descrilje the 
relationships arong a nunber of process variables associated with the costs 
of operation, md output variables measuring the clients' gain in Jaiwledge 
and satisfaction. Strongest relationships were hetwmn process variables sudi 
as tiie^Tfiftbe^^.Q£^fcitations printed and price , betvsr'oen pricfe and satisfaction, 
and be^een the amomt leamed and satisfactien* The negative rjlaticnship 
be ween price and satis factlOTi, whidi did not apply t^^at is faction with the 
blbliograpHy and nuiterials tlieir^elves, suggests that services which charge rmst 
be aware tMat clients are rating tlieir service as well as their oroduct* 



Future research is neodfid to dlscovisr variables that dDteimm^the amimt 
a client Ican^s from a search, since this informatlcsn wuld give the search 
analyst a better mderstanding of when and hw a seardi should be conctuctcd. 
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Items llsai in Six ^tisfaction aihscales 



^estion; Please indicate ymr satisfaction with tte fallowing el^Gnts 



SATISl - Riblieity Material airf Direct ions 

a. Convenience of arrangsrients and adet^cy of diroctions 



c. Adequacy of directions for orderii^ materials 

SATIS2 - Convenience and Ffelpfulmss 

a. Convenience of arrangCTients and adequacy of direction 

b. I%lpfulness of the search arialyst 

SMIS3 ^ Timeliness of Serviue 

a* Time taken to deliver bibliography 

Tiine taken to deliver naterials from TOS 

c. Time taken to deliver rater ials from RISD 

a\TIS4 - Quality of T^hnology . 
a* Length of bibliography 
b* Readibility of microfiche copies 
c» Availability of microfiche readers 



/ 



of ycwr nrsn s^rch* 



b. Accuracy and conrpreh^raiv^ess of pwhlicity materials 



S^TISS " Value of Bibliograpliy and ^laterials 




b. Materials created via bibliography 



S/VTIS6 - Overall Satisfaction 



la^ 2b p 3a s 4a, Sa 



&mlc scores wera adjust^ for the nimhef of itms by dividii^ total 
-m^le XQTQs by the numher^of itgns, Pcalei Low ■ l* medltin * 2; 
high ^ 3| H\ - 4. 



I 
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TABLF I 



Process and Outrtit VariaMes Vscd in Search Somco T^luntiQn 



Itm noscrintion and rodinf? 



1 mSTACT** In porson - I 
Other - n 



2 SfSTm 



LIS mara - n 



3 TliRhi\Rni?n ^TTOrounci time in ckys betveen receipt of request nrd 

on-line searching 

4 CNCTDtn Connect-time in minutes 

■ S STMTDIT! Interviw and strmtc^ tiro in minutes 

6 crrmiNT NLmber of citations printed - 

7 reiCE r¥ice charged in dollars 

8 A^niRDTC ibw nnjch did you leam about your topic as a result of 

V the search? r- 

(1) Nothing or very little 

(2) „ Some 

t3) A great deal 

9 S\TIS6* fVerall satisfaction 

10 SMISl Satisfaction with FHibllcity Materials nml nircctions 

11 .S\TIS2 &tisfactl6n with Convenieree and Helpfulness 



ERIC 



12 5i\TIS3 \ .^tisfaction with TinoUness of Senrlce 



13 S^Trs4 fetisfaction with nuali^ of Technolngy 



14 S\TTS5 Ritisfaction with Value of Bihliography 



^^**\^ariahles 1 through 7 rrcordrf by. s^rch aralyst on data sheets. 
Variables 8 through IS indlcatri by client on fJsor Pi^luation 
Qiiestlonnairep ^ 

*See Appendix A for complete list of satisfnction itefis and 
construction of siihscales. 



s: . 

CorrelatiQn >*atri^j Means and Standard novlations 
for \^ariablos in Process hfodel Fjrplaining 
Overall *%tlsfaction with Serviee 

^1 ^2 h ^4 ^ ^ i 

sYsrm .nos 

X^ CNCTM .357' -,118 .OPS 

STO^TIME .474 -.295 .465 *433 

Xg CimiNT .296 -.032 .117 .530 ,341 

X^ PRICE .086 .070 .151 .311 .309 .621 

Xg A.^fn^nTt: .02s -.097 ^.ooe .oss .044 ai7 .039 

%g miS6 ' .036 .003 .060 .079 .038 .120 -.040 .394 

M^n .531 .193 4.31 24.2 46.2 104 28.3 2.19 2.61 

Std.D0v. .501 .396 4.26 14.5 ,26.3 111 ill 0.62 .32 

N * 148 



p .01 
* p <.0S 



Path Cpefflcients for Process hbdcl Explaining Sidjscale feores for 
SatisfactiCTi witJi fendce 



PatJi * Statistic Varid^le 

SATISl SATIS2 SATIS3 SkTm SATISS SATI^ 



1. 


COSTACT'SYSTBI 


* i£ 


.03 


-,01 


-.23 . 


.16 


.06 


,00 


2, 








*03 


19 


- ,19 


-.06 


05 


3. 




-23 


. 27** 


-,29** 


,01 


-,32** 


- 34** 


- 30** 


4. 


CCTBE-OTJTACT 


P41 


,29* 


.32** 


28 


.20 


, 




S. 


STOATOffi-COOTACT 


P51 


,39*^ 


.41^* 


.21 


,34** 


,40** 


.47** 




CITPRINT-QJCTM 


P64 


,47** 


,46** 


,44* 


,62** 


,S6** 


.47** 


7. 




P65 


,15 


,14 


.21 


-,04 


,05 


.14 


8, 


PRICE'CITPRINT 


P76 


,66** 


,61** 


,29 


,57** 


,50** 


.62** 


9 




Hz 






' .^j 


■ u*? 


^ 1 A 

* ^H- 




10. 


A^ni^Drrc-sm\Ti>E 


Pss 


-,06 


-.01 


-.04 


-,08 


-,03 


-.03 


11, 


A^f^LHDTC'CITPRINT 


Pee 


.15 


.15 , 


,02 


.01 


,05 . 


.14 


12. 


SATIS'SYSTB! 




.04 


.04 


,21 


,32* 


-.05 


.09 


13. 


SATIS-TimNARVD 




,03 > 


.05 ' 


,08 


,26* 


,09 


.09 


14. 


SATIS-CITPRINT 


P96 


-,13 


.14 


,41* 


-,01 


,03 


.18 


IS, 


SATIS-PRICE. 


P97 


-,24* 


-.22* 


-,39* 


-,11 


-,06 


-.20* 


16, 


SATIS'AmROTC 




-*08 


.09 


.43** 


.11 


.67** 


.38** 




N 




148 


157 , 


31 


70 


139 


145 • 




X 




2,60 


2. 75 


2,40 


2,59 


2,32 


2.61 




s 




*42 


■*2 


.45 


,37 


.67 


.32 



** p < ,01 



* p > \QB 













^^^^^^^^^^ 
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FIGUIili 1, rnformation System Moclel 
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